The tremendous growth in both speed and reliability of today's optical networks has enabled the adoption of new communication paradigms that allow distributing human resources, databases, and operations across the entire globe. This revolution is dramatically increasing productivity and growth and laying down the foundations of future global societies.
As for any new revolutionary technology, this changing landscape also poses great threats to both our security and our new business processes. Emerging needs in network engineering, and network security require advanced monitoring of the data packets traveling through the networks. This monitoring is necessary to (1) form statistical models of the type of traffic utilizing the networks to aid in the design and optimization of the networks, (2) detect anomalous use of the network resources for improper attempts to violate access control policies and (3) gather intelligence on crime and national defense.
In addition to monitoring, defensive mechanisms are also necessary to quickly adapt and protect information systems to changing requirements or identified security threats. These needs go beyond the classic router and firewall functions of the OSI layers 3 and 4 and require deep packet inspection and classification policies based on any portion of the packet including its payload. These requirements are especially relevant for applying policies at the gateways between different administrative domains or in distribution centers where threats and instability can quickly spread among of 1000's of users.
The pace of increase of the communication speeds has created a lag in the ability of network monitoring devices to observe, categorize and analyze the traffic in real time. In particular, monitoring applications that require the analysis of the payload of the data packets are suffering today from limitation in performance that forces best-effort types of modality. This limitation prevents network operators from aggregating traffic streams for analysis and forces expensive and inaccurate monitoring solutions.
The enforcement of traffic filtering and rerouting policies are also affected by the increase in speed. Typically, conventional network computing power, originally designed for packet forwarding, is retrofitted and adapted to meet high-speed filtering requirements. This makes current defense mechanisms, inefficient, non-scalable and difficult to operate.
Current network processing technology is based on highly specialized microcontrollers designed to perform operations common to packet forwarding functions. These microcontrollers typically provide hardware support for (1) decomposing and reassembling packets, (2) look up tables for making routing decisions (typically achieved through the use of content addressable memory CAM), and (3) multithreading to be able to time-multiplex the processing of multiple packets.
Such microcontrollers have also been used to perform signature matching. Such an implementation is shown in FIG. 1. Referring to FIG. 1, a number of network processors 103 compare the incoming packets of data stream 104 to rules 102 stored in memory 101. Data stream 104 is load balanced across processors 103 in order to allow each processor enough time to compare the incoming data to rules 102 encoded into memory 101. The width of the input data path is typically fixed for each processor type and the synchronization of the matching operation is achieved with ad-hoc external circuitry. This architecture has the advantage of reusing and retrofitting existing network processing technology thus reducing cost but has considerable scalability constraints.
The problem with such current hardware-based signature matching implementations is both (1) an increase in the data stream input rate and (2) an increase in the number of signatures requires the addition of processors. This means that scaling can only occur with an increase in cost and synchronization overhead.